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ABSTRACT 

The key question in the research reported is whether 
a high investment in the design of written materials for use in 
distance education is worth the effort. Specifically, the study 
investigated the actual use of embedded support devices (ESDs) by 25 
students at the Open University of the Netherlands in a distance 
education governmental law course. Students did use ESDs, vn th 60 to 
80Z of students using nearly all the ESDs. Women were more likely to 
use planning ESDs, orienting ESDs, and processing ESDs on a deeper 
level than men. Students w.fth higher prior education also seemed to 
use ESDs on a deeper level, as did students with less prior knowledge 
of governmental law and those using study guidance. No interrelations 
were found between cognitive student characteristics and the use of 
ESDs. High investment in the design of written materials seems v»orth 
the effort, but student characteristics should be taken into account 
in writing ^.he materials. Six tables present study data, and three 
appendixes provide supporting information. (Contains 14 references.) 
(SLD) 
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OTIC RESEARCH REPORTS. 



Hie Open University is responsible for developing and offering open, higher distance education in which special, attention is paid to innova- 
tions in educational technology. The research in this field is concentrated in "OTIC", that is the Centre for Educational Technological 
Innovations (Onderwijs Technologisch Innovatie Centrum). 

OTIC is also engaged in running project! for other institutes. Here the Centre makes use of OTIC's knowledge and experience acquired in 
research and development 

The series of OTIC Research Reports consists of publications of the OTIC researdj projects and aims mainly at an audience of fellow 
researchers. 



RESEARCH PROJECT 'SUPPORT BY INDIVIDUAL GUIDANCE'. 

The SIG project 'Support by Individual Guidance' (Begeleiding door Individuele Ondersteuning - BIO) focuses on the opcrationalisation of 
new approadies towards student support in distance education and open learning settings. 

The main objectives are in the field of flexible organisation of support, incorporation of support into learning materials, optimizing the link 
between prior knowledge state and the support needs, and the development of more flexible (and automated) approaches towards guidance 
oriented evaluation. 

This project is embedded in the larger research domain 'intake, guidance and evaluation'. This helps to build on the outcomes and 
experiences of e.g. the prior knowledge project, projects in relation to automated evaluation , etc. This multi-disciplinary and multi-focus 
approach is necessary to pursue the wide diversity of the project objectives. 
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Het feitelijk gebruik van ingebouwde begeleidingscomponenten in schriftelijk leermateriaal 
voor afstandsonderwijs 

Nederlandstalige saaienvatting OTIC-onderzoeksrapport 54 



Inleiding 

In dit rapport wordt verslag gedaan van de empirische fase van een onderzoek naar het gebruik van 
ingebouwde begeleidingscomponenten in schriftelijk afstandsonderwijs door studenten van de Open 
universiteit. Na een literatuurstudie en theoretische uitwerking (Valcke e^, 1991; Valcke en Martens, 1992) 
werden meer dan veertig verschillende ingebouwde begeleidingscomponenten onderscheiden (nader te 
noeinen: ESD, Embedded Support Devices); Een groot gedeelte van deze begeleidingscomponenten wordt 
momenteel bij de uitwerking van Ou-cursusmateriaal gebruikt (bijv. inhoudspagina's, stiuctuurpagina's, 
advance organizers, voorvragen, voorbeeldtoetsen, illustraties, margeteksten, schema's, opdrachten etc.). 

Uit de literatuur blijkt dat de theoretische onderbouwing voor het aanwenden van deze ESD slechts beperkt 
of niet bestaande is. Aan de ESD worden fimcties en effecten toegeschreven die niet ondersteund worden 
door onderzoeksbevindingen. Bijkomende bezwaren bij vroeger onderzoek zijn ook dat dit niet is uitgevoerd 
in de context van afstandsonderwijs en dat eerder naar produkteffecten dan proceseffecten is gekeken. 
Ondanks de gebrekkige theoretische en empirische onderbouwing wordt er aan de Open universiteit veel 
geinvesteerd in het implementeren van ESD in schriftelijk studiemateriaal. 

De hoofdvraag van dit onderzoek is dan ook of deze investering effectief is, waarbij we voor het 
beantwoorden van deze vraag uitgaan van de afgeleide vraag : "Hoe gaan studenten om met de ingebouwde 
begeleidingscomponenten ?". 

Na een selectie van onderzoeksmethoden (Valcke en Martens, 1992) is gekozen voor de interviewmethode, in 
combinatie met een schriftelijke vragenlijst. 

Nul-hypothesen 

De volgende nulhypothesen werden naar voor geschoven : 

Studenten gebruiken geen ESD (kwantitatief gebruik) 

Er is geen verschil tussen het aantal studenten dat ESD oppervlakkig of op een diepgaande 
manier gebruikt (kwalitatief gebruik). 

Er is geen verband tussen algemene studentkenmerken (zoals intrinsieke motivatie, geslacht, 
algemene motivatie en locus of control) en het gebruik van ESD. 

Er is geen verband tussen cognitieve studentkenmerken (zoals algemene studeerstrategie, 
gebruik van exteme begeleidingsvormen, ervaring met afstandsonderwijs) en het gebruik 
van ESD. 

Op basis van de verzamelde onderzoeksgegevens konden twee extra hypothesen gecheckt worden : 

Het gebruik van ESD heeft geen effect op studiesucces (tentamenscore en aantal tentamenpogingen). 
Er is geen verschil tussen de theorevisch verwachte functies/effecten van ESd en de functies en 
effecten die studenten spontaan aan de ESD toeschrijven. 

Procedure 

De onderzoekspopulatie bestond uit 25 rechtenstudenten die de cursus 'Inleiding tot het bestuursrecht' van de 
Open universiteit bestudeerden en in de loop van het half jaar voor het interview aan het eindtentamen 
hebben deelgenomen. 

Deze studenten komen uit een groep van 51 studenten die bereikbaar waren vanuit de studiecentra Sittard, 
Eindhoven en Venlo. Het profiel van de 25 studenten was representatief voor het type studenten dat de 
cursus 'Inleiding tot het bestuursrecht' bestudeerL 



Het onderzoek was gebaseeid op een analyse van de antwoorden van studenten op gestructureerde interview- 
vragen. Alle interviews werden opgenomen op band. Daamaast zijn tijdens het interview notities gemaakt en 
zijn de aantekeidngen van studenten in hun cursusmateriaal op intensiteit van gebniik onderzocht 
(bijvoorbeeld: gebniik van kleunnarkeringen en onderstrepingen). 

Omdat het interview sterk gestructureerd was konden de meeste antwoorden direct herleid worden tot 
specifieke antwoordcategorieen bij in het totaal 499 interviewvariabelen. Deze gegevens werden als ruv/e 
data ingevoerd in een computerbestand. Vervolgens zijn de interviewvariabelen gerubriceerd, 
getransfonneerd, bewerkt en herleid tot 191 onderzoeksvariabelen. Ten aanzien van ESD-gebruik werden in 
het onderzoek 23 ESD nader onderzocht. Het gebniik voor elke BSD afzonderlijk werd onderzocht en van 
clusters binnen deze ESD (verwerkende, toetsende, orienterende en plannende ESD). 
Met behulp van het statistisch pakket SPSS"^ zijn de data verder geanalyseerd. 

Analyses 

Met behulp van descriptieve gegevens (percentages en gemiddelden) is in kaart gebracht door hoeveel 
studenten ESD gebruikt worden en hoe ze gebruikt worden. 
Geconcludeerd werd dat: 

- ESD veel gebruikt worden, 

- ESD overwegend op een diepgaande manier gebruikt worden. 

Met behulp van loglineaire analysetechnieken is nagegaan welke verbanden er bestaan tussen het gebniik van 
ESD (kwantitatief en kwalitatief) (2*23 variabelen) en algemene en cognitieve studentkenmertcen (30 
variabelen). Er is gebniik gemaakt van de SPSS'" HILOG procedure om modellen te toetsen. Modellen 
werden gescreend met behulp van hierarchische loglineaire analyse (backwards st^wise). 
Geconcludeerd werd dat er duidelijke verbanden zijn tussen algemene studentkenmeiken en (diepgaand) 
gebniik van ESD: geslacht, opleidingsniveau, vooikennis van bestuursrecht en gebniik van 
studiebegeleiding. Vooral hoogopgeleide, intrinsiek gemotiveerde vrouwelijke studenten met veel studie- 
ervaring gebruiken ESD meer op een diepgaande manier. 

Er was een negatief verband tussen voorkennis en het gebniik van 'toetsende ESD'. 

Er werd geen relatie gevonden tussen cognitieve studentkenmerken en het gebniik van ESD. 

Opgemerict meet worden dat de proefpersonen groep sterk homogeen was (met betrekking tot ESD-gebruik), 

hetgeen de kans op het yinden van significante verschillen mogelijk verkleind heeft 

In een volger.de reeks loglineaire analyses is nagegaan hoe de tentamenuitslag en het aantal 

tentamenpogingen zich verhouden tot ESD gebniik. Er is vastgesteld dat zowel de tentamenuitslag als het 

aantal tentamenpogingen samenhangen met de algemene studeerstrategie. Een diepgaand ESD gebniik hangt 

samen met hogere tentamenuitslagen en minder tentamenpogingen. Er zijn ook verbanden met onder andere 

het opleidingsniveau en ervaring met afstandsonderwijs. 

In een laatste reeks analyses is onderzocht wat het verband is tussen theoretisch verwachte fiincties en 
effecten en spontaan door studenten genoemde functies en effecten van ESL. Om dit na te gaan werd bij de 
vragen die handelden over het specifieke gebniik van ESD altijd eerst een zeer open en niet-sturende vraag 
gesteld ("Wat vindt u van dit soort informatie?"). 

Op deze manier wilden we voorkomen dat antwoorden in een bepaalde richting gestuurd zouden worden. 
Aansluitend werden meer gerichte vragen gesteld waarin wel de functies van ESD werden genoemd ("Heeft 
de inhoudspagina u geholpen bij het orienteren in het cursusmateriaal ?"). 

Onderzocht is of de spontaan gegeven antwoorden overeenkcmen met theoretische verwachtingen, gebaseerd 
op het model dat is uitgeweikt in Valcke en Martens, 1992. 

Er bleek een duidelijke discrepantie te zijn tussen de theoretisch verwachte functies en effecten en de 
spontaan genoemde fiincties en effecten van ESD. Studenten schreven niet bepaalde functies en effecten toe 
aan ESD terwijl dat op grond van de theorie weJ. verwacht werd. Ook noemden zij andere functies en 
effecten van ESD die niet verwacht werden op basis van de theorie. 
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Een mogelijke veridaring hiervoor is dat hst niet te verwachten is dat studenten in staat zijn om spontaan alle 
mogelijk effecten van ESD te noemen. Ook is het mogelijk dat studenten zich niet bewust zijn van de 
mogelijke fiincties van ESD. Een andere interpretalielijn zou kunnen zijn dat studenten wel in staat zijn 
functies en effecten toe te schrijven aan ESD, maar dit doen binnen hun eigen momentane studiebenadering 
en -strategie. Deze verschillen van student tot student 

Conclusies ten aanzien van de nul-hypthesen 

De volgende nulhypothesen kunnen warden verworpen: 

Studenten gebruiken geen ESD (kwantitatief gebniik) 

Er is geen verschil tussen het aantal studenten dat ESD oppervlakkig of op een diepgaande 
manier gebniikt (kwalitatief gebniik) 

Er is geen veiband tussen algemene studentkenmeiken (zoals intrinsieke motivatie, geslacht, 
algemene motivatie en locus of control) en het gebniik van ESD 
De volgende hypothese kan niet verworpen worden : 

Er is geen verband tussen cognitieve studentkenmeiken (zoals het gebniik van exteme 
begeleidingsvomien, ervaring met afstandsonderwijs, etc.) en het gebniik van ESD. 

Discussie, extrapolatie van de onderzoeksresultaten en vervolgonderzoek 

De uitgangsvraag : "Loont de uitgebreide didactische uitweridng van Ou-cursussen op basis van he gebniik 
van ingebouwde begeleidingscomponenten, de moeite ?" kan naar onze mening positief beantwoonl wonlen. 
De ondrazoeksresultaten geven echter aan dat bij deze uitweridng meer rekening kan/moet gehouden worden 
met het 'type' student dat men voor ogen heefl. Blijkbaar hebben niet alle studenten dezelfde behoeften aan 
ESD als andere. Daarbij komt de vaststelUng dat het toeschrijven van fiincties en effecten van ESD tussen 
door studenten vesichilt van wat theoretisch VCTwacht wordL 

Dit laatste, gecombirieCTd met de eerste vaststelling leidt ons inziens tot een argumentatie voor e^ flexibele 
cursusproduktie enerzijds en een flexibel doorlopen van de Ou-cursussen door een student anderdjds. In 
concrete zou dit kunnen inhouden dat bijv. verschillende versies ontwikkeld worden van 66nzelfde Ou- 
cursus, afhankelijk van het soort gebniikers dat men voor ogen heeft (bijv. diplomastudenten versus 
professionaliseringsgerichte gebniikers). In een ander perspectief kan gedacht worden aan een 
geautomatiseerde leeromgeving die geent is op een Ou-cursus. Een student zou deze leeromgeving kunnen 
gebruiken om tijdens de eerste fase van het bestuderen van de cursus keuzes te maken in verband met de 
inhoud en de begeleidingscomponenten. Pas dan zou een - aangepaste - Ou-cursus afgednikt worden. Een 
andere extrapolatie van de onderzoeksgegevens kan inhouden dat studenten getraind worden in het gericht 
gebniik van ESD. 

Een relativering t.a.v. de gevonden onderzoeksresultaten is hier op zijn plaats. De karakteristieken van de 
onderzoeksgroep, de aard van de cursus die in het ondeizoek werd betrokken en de gevolgde 
onderzoeksmethode (interview) kunnen de onderzoeksgegevens beinvloed hebben. Hierdoor is een directe 
generaliseerbaarheid van de bevindingen niet gepast 

Toch kunnen we stellen dat de gevonden resultaten betekenisvol zijn en blijven. De gevonden resultaten 
stemmen bijvoorbeeld overeen met resultaten vroeger onderzoek dat door het onderzoeksteam opgezet werd 
bij dezelfde rechtencursus. Nog belangrijker i;i dat de gevonden kwantitatieve en kwalitatieve gegevens m.b.t 
ESD gebniik en de samenhang met bepaalde studentkenmericen ook bevestigd worden door 
onderzoeksresultaten bij een niimere Ou-populatie (cf. de jaarlijkse JOO-onderzoeken). Meer onderzoek blijft 
echter wenselijk. Bij dit vervolgonderzoek zouden altematieve onderzoeksmethcden moeten gebniikt 
worden. Daamaast is ook onderzoek bij andere cursussen, inhoudsdomeinen en studentenpopulaties 
wenselijk. 
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The actual use of embedded support devices in self -study materials by 
students in a distance education setting 



ABSTRACT 

The key question dealt with in the research reported here can be stated as follows : "Is the high investment in the 
design of written study materials for use in the distance education setting, worth the effort ?". The written study 
materials, used at the Dutch open university (Ou) imply a high investment in the didactical elaboration of the smdy 
texts by embedding a large variety of support devices. The actual use of these embedded su{^rt devices (BSD) is 
questioned in this study. To answer the question, our research focuses on quantitative and qualitative aspects of ESD 
use. 

The following scientific hypotheses were specified: 
Quantitative hypothesis: 

1. Students make no use of ESD. To check this hypothesis descriptive statistical techniques are used. 
Qualitative hypotheses: 

2. There is no difference in the number of students making surface level or deep level use <^ the ESD. 
Checjdng this hypothesis implies descriptive statistical tedmiques. 

3. There is no interrelation between general student characteristics and the use of ESD. 

4. There is no interrelation between cognitive student characteristics and the use of ESD. 
These last two hyixjtheses are checked using a multi-way ftequency analysis. 

The information gathered, is also helpful to focus on two other hypotheses : 

5. Student study success is not i\:Iated to ESD use. 

6. There is no difference between theoretical expected and spontaneously mentioned functions and effects of 
ESD. 

The researdi setup was based on an in-depth analysis of responses to stmctured interviews. The research sample 
consisted of 25 Ou-law students who attended the final examination in 1991 for the course "Introduction to 
govenmiental law". 

Analysis of the research data indicate that hypotheses 1 and two can be rejected. Students make use of ESD; neariy all 
ESD are used by more than 60% to 80% of the students. Furthermore students make deep level use of the ESD. 
Also the third hypothesis can be rejected. Interrelations between general student characteristics and the use of ESD are 
also found. Women seem to be more' likely to use planning ESD, orientating ESD and processing ESD at a deep level 
Students with higher prior education levels seem to use ESD at deep level to a higher extent. Students with less prior 
knowledge of governmental law and students using study guidance use testing ESD more at a deep level 
The fourth hypothesis cannot be rejected; there are no interrelations found between cognitive student characteristics and 
the use of ESD. 

To get a better view of the 'type* of students using ESD, the intenelations between generd student characteristics have 
been analyzed. Especially higher educated (and intrinsic motivated) female students with a lot of study experience and 
less prior knowledge make deep level use of ESD. 

When linking the use of ESD to study success Qiypothesis 5) the following picture arises : Students using ESD at a 
deep level are not only more likely to have a higher score on the examination, they also need less examination attempts. 

Looking at the results of the comparison of theoretical and spontaneously mentioned fifflctions and effeas of ESD 
(hypothesis 6), the results reveal that students indicate functions that are not theoretically expected. Smdents seem to 
ascribe their own functions and effects to ESD. 

The main conclusion of this research is that the high investment in the design of written smdy materials is worth the 
effott, but that it is impottant to take account of student cfamcteristics when implementing ESD in course materials. It 
can for instance be argued to incoipotate into the materials a 'training' about the use of the ESD. Moreover, these 
results link up with the growing tendency to develop more flexible Ou-couises that ate adaptable to individual student 
needs. The latter implies that the production of written veniions of the study materials is to be redirected towards the 
development of automated interactive learning environments. 
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1. INTRODUCTION 



In an earlier OTIC research report (Valcke & Martens, 1992), the theoretical base and an overview of 
research ^proaches was discussed in relation to the SIG-project "Support by bidividual Guidance". The 
SIG-project is set up by researchers of the Centre of Educational Technology and Innovation (OTIC) of the 
Dutch open university. 

The Dutch open university (Ou) organizes and supports university level education at a distance. This implies 
that the way learning and instruction is organized is adapted to particularities of this setting. Of outstanding 
importance in this perspective is the production of high quality study materials. Up to now, these materials 
are still mainly text-based. The SIG-project focuses on the quality of these written study materials. 
A key question of this project is : "Is the high investment in the design and elaboration of written study 
materials for use in a distance education setting, worth the effort ?". Is the elaboration of the study text with 
a high amount of embedded support devices (structure pages, content pages, indexes, questions, tasks, 
schemes, examples, etc.) a functional and necessary elaboration that meets the demands of the Ou-studwits 
(Embeo '^d Support Devices will &om now on be abbreviated as ESD). 

To answer this question, we first specified a theory about fte potential functions and effects of embedded 
support devices in written study materials. This theoretical, hypothetical construct is considered as the 
starting point from where to look at the actual use of these ESD (embedded support devices) by students 
when they study fte course materials. This point of view is helpful to obtain : 

quantitative information about the question whethw or not ESD are used and to what extent; 
qualitative infonnation about the extwit the ESD are used by students (e.g. deep level or surface 
level). In which way(s) use students the ESD ? Related to the latter, a close look at variations in use 
of the ESD in relation lo personal characteristics of the students is extremely helpful. In this way we 
obtain details about the necessity to vary the elaboration of ESD and study materials according to 
individual differences. This information is a necessary source to make study materials flexible and 
adapted to individual needs. We further question whether the actual use of the ESD reflects the 
theoretical, hypothetical functions and effects of ESD ? Finally wecan also link the use of BSD to 
study success. 

This research rqrort starts with a short sununary of the theoretical base and background of our research in 
order to describe the research hypotheses and research variables. Chapter 3 focuses on the description of the 
research setup. Attention is paid to the selection of the research sample, the analysis of the Ou-course 
"Introduction to governmental law" to define an operational subset of ESD, the interview and the interview 
procedure. An analysis of the research data based on descriptive statistics to draw conclusions at the 
quantitative level is supplied in ch^ter 4. Chapter 5 contains a multi-way fi^uency analysis of the research 
data to deduce conclusions at the qualitative level. In chapter 6 we discuss the students score on the 
examination and the number of attempts to pass the examination in relation to the 133 of ESD. Chapter 7 we 
confront the functions and effects of ESD as reported by the students with the theoretically expected 
functions and effects of ESD. 

Ch^tCT 4 and 5 each comprise a discussion of tiie research results. Ch^ter 8 goes beyond a local 
discussion of the results by positioning tiie research findings against results of otiier research. At tiie same 
time tiie relevance of tiie results to (re)direct educational practice at tiie Open university are debated and 
directions for future research are derived. We end tiiis report witii a concluding discussion in ch^ter 8. 



ERIC 



11 



2. 



RESEARCH BACKGROUND, HYPOTHESES, AND VARIABLES 



2.1 Theoretical base and background of the study 

As stated in the introduction, our research focuses on quantitative and qualitative aspects of the use of 
embedded support devices in distance education study materials. 

To summarize in a short the theoretical base and background for ?his research, we repeat the following : 

Embedded support devices (ESD) are defined as formal and/or content-related add-ons, extensions 
and elaborations of the learning content in the written study materials (objectives, prior knowledge 
state tests, structure pages, content pages, indexes, questions, tasks, examples, illustrations, 
typography, etc.). They aim at scaffolding the learning process by supporting orientation, planning, 
processing and testing aspects of the individual learning process. 

From earlier research in relation to ESD we learn that this research can be criticized since 
there are important theoretical and methodological weaknesses. This is especially true iif 
the results of this research body is to be used in the context of distance education study 
materials. 

A theoretical base has been elaborated for our research that is founded on a cognitive 
processing approach taking account of the learning process, the reading process, the task 
environment and characteristics of the study materials. 

The theoretical base has been used to describe potential functions/effects of all types of 
ESD. 

After reviewing methods to research the use of ESD by students in a distance education 
setting, we decided to opt for the use of an approach based on the use of structured 
interviews. 

2.2 Deriving hypotheses 

Since the key question of this project is : "Is the elaboration of the study text with a high amount of 
embedded support devices (structure pages, content pages, indexes, questions, tasks, schemes, examples, etc.) 
a functional and necessary elaboration that meets the demands of the Ou-students ?", we have to focus on 
quantitati' t and qualitative aspects of ESD-use in relation to student characteristics. The hypotheses we 
derive from this research question show a narrowing down from quantitative features of general ESD-use to 
qualitative features of ESD-use related to student characteristics. As such, two sets of hypotheses are 
defined : 

Set 1 : quantitative hypothesis 

1. Students make no use of ESD. 

Set 2 : qualitative hypotheses 

2. There is no difference in the number of students making surface level or deep level use of the 
ESD. 

3. There is no interrelation between general student characteristics and the use of ESD. 

4. There is no interrelation between cognitive student characteristics and the use of ESD. 

The information gathered and the theoretical statements about functions and effects of ESD are also helpful 
to state additional hypotheses about ESD-use : 

J. Student study success is not related to ESD use. 

6. There is no difference between theoretically expected and spontaneously mentioned functions and 
effects of ESD. 
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2.3 Research variables 



When stating the hypotheses in relation to our research question we already - implicitly - pre-defined sets of 
variables that have to be researched : 

variables in relation to ESD-use; 

variables that look at more detail to quality features use of ESD-use; 

variables that are helpful to describe personal characteristics of the students in our 

research sample; 

variables that are helpful to describe general cognitive characteristics of the students in 
the research sample. 

A detailed overview of the research variables v/ill be discussed in part 3.6. These variables are derived 
from the interview variables reviewed in part 3.4. 

3. RESEARCH SETUP 



3.1 General description 

As argued in Valcke & Martens (1992), we opt for a research setup, based on the analysis of data inferred 
from interviews. Since we prefer a research set up with high ecological validity, we are well aware of the 
difficulties in controlling for a multitude of variables influencing the research data. Therefore, much 
attention was paid to the selection of the research population, the analysis of the course "Introduction to 
governmental law", the construction of the interview and the procedure to derive research variables from the 
interview. Nevertheless, we have to keep in mind, when deriving conclusions from our research results, that 
the research data can be biased due to the research procedure. 

3.2 Research population 

The SIG-pioject links its empirical research to courses in the law domain. For this research, we focused on 
students taking the course "Introduction to governmental law". 

From the Open university data base about student enrolment and exams, we extracted the complete list of 
students who participated in the 1991 examinations for this course. We selected these students, to be sure 
that students really studied the course and used the course material. From this list we selected 51 students 
who live close to three specific study centres of the Open university (Sittard, Eindhoven, Venlo). 

The selection of 'students taking examination' can question the representativeness of our research sample : 
Do we not omit students who do not take examinations due to shortcomings in tlie written study materials ? 
Earlier research at the Dutch open university reveals that 45% of students subscribing for introductory 
courses (the 'type' of students we focus at) do not or do not plan to participate in examinations. But, as to 
the rationales for not attending examinations, refined analysis reveals that not taking examinations is not 
related to the quality of the study materials (Kempkens, 1987, p.60). Tlie findings of more recent research 
with Dutch Ou-students confirm the results of Kempkens : Ou-students, independent of the fact whether 
they attained a certificate or not, highly appreciate the content and elaboration of the written study materials 
(Boon et al, 1991a, p. 12). 

Kempkens (ibid, p. 20-25) also analysed drop out in relation to examination participation and study success 
at other open or distance education universities. She finds comparable conclusions, sustaining the hypothesis 
that examination participation is not related to or has certainly not a primary significant intenelation with 
quality features of study materials. 

The students received a letter to describe the research project and to ask their participation in the research. 
Two weeks later all students were contacted by phone. A protocol was prepared to structure this phone call. 
This resulted in 25 students who were willing to participate. A letter was sent to these students, confirming 
the date and study centre location for the interview. 
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The final sample of 25 students consisted of 19 male Mnd 6 female students, a ratio which is representative 
for the population participating at the law course "Incoduction to governmental law". 
The educational level of this students was rather high: only 1 student had a low educational level (elementary 
school); 12 students had a medium educational level (secondary school); 12 students had a high education^ 
level (higher education). 

The age level of the students was divided as follows: 



Further data about the student characteristics of the research sample are displayed in appendix 1. In general 
it can be stated that the research sample is a representative subset of the student population taking the course 
"Introduction to governmental law". Despite this fact, the selection of this research sample can influence the 
extent to which we can or might generalise our reserach conclusions. Are law student different from e.g. 
student taking social science courses or studying cultural topics ? We have not analysed our reserach sample 
in relation to the total Ou student population. Therefore, cation will be taken when generalising conclusions. 

3.3 Analysing the course "Introduction to governmental law" : a subset of BSD 

As stated earlier, we analyze study behaviour of students taking the course "Introduction to governmental 
law". This is a first level course designed for the law scientific diploma line. 

Since we are interested in the actual use of ESD by students, this implies a detailed analysis of this course. 
Learning unit 20 of this course was selected to be used in the interview. This unit was selected since the 
course developers and tutors indicated this was considered as a difficult unit. Morsovw, this unit comprises 
a large subset of the ESD we focus on. Each page of learning unit 20 was analyzed to detect where ESD 
were embedded. This selection was done making use of categories for ESD that are described in Valcke et 
al, 1991. Twenty one different ESD are embedded in learning unit 20. Of two of these ESD, two examples 
are used in the interview, totalling the number of ESD used in our research to 23. Table 1 presents a list of 
these ESD. 

3.4 The interview and interview procedure 

3.4.1 Interview elaboration and structure 

Seven clusters of interview variables' were defined : 

Personal student characteristics : g<}nder, age and educational level; 

Motivation level : for studying at tlie Open university, for studying law and studying this 

specific course; 

Prior knowledgel experiences : with studying at the Open university, studying law and 
studying the specific course "introduction to governmental law"; 
Task environment variables : use of specific support provisions; 

Learning strategy : general learning strategy, learning strategy in relation to specific course, 

learning strategy learning unit 20, use of the 23 ESD; 

Metacognitive variables : use and development of metacognitive strategies; 

Time structuring : time used to study, time planning; 

A first version of the interview was composed and discussed with a team of OTIC-researchers. Also a try- 
out with colleagues provided materials for further discussion. Especially the length of the interview posed 
problems. Therefore, it was decided to split up the interview into two parts. Some questions were excluded 
due to the time-constraints. 



It was not necessary to define interview variables in relation to all researdi variables since we could rely on 
information already present in the Open university student data base (e.g. number of examination atten^ts, 
examination score, etc.). 



18-30: 
30-40: 
40-50: 
>50: 



5 
9 
7 
4 



5 



ERIC 




Part I was sent to the students a week before the interview appointment in the study centres. The students 
were expected to fiU in this part at home. This part consisted of questions related to the first four clusters of 
variables explicitated above. 

Part n was elaborated as a very rigid protocol in which the focus is on the variables related to the three last 
clusters of variables. Each question in the protocol is unambiguously connected to one particular variable. 
Along this interview protocol, the focus of the discussion is continuously narrowed down fiom the general 
individual learning strategy of the student to his/her specific use/^proach of ESD when studying learning 
unit 20 of the course. In relation to each single HSD of learning unit 20, a double strategy was observed. 
First we provoked spontaneous reactions of the students that indicated their opinion about and use of the 
specific ESD. Next, we faced them with questions/statements that reflect hypothetical functions/effects we 
derived from our theory. They could indicate whether or not they agreed with these questions/statements. 

3.4.2 Interview procedure 

The students were expected, as stated above, to fiU in part I of the interview at home. 
During the interview, which was all set up during a period of 2 weeks, the focus was on the 
questions of part U. The students were invited individually for this interview session in a study 
centre. They were asked to bring their own study materials with them. Each interview lasted about 
90 to 120 minutes. The interviews were recorded on audio tape. The researchers took note of key 
words during the interview. 

The introduction to the interview stressed the non-evaluative nature of the interview, 
although this was already mentioned in the letter and the phone call preceding the actual 
interview ses. 'on. 

According to the interview protocol, students were allowed or demanded to go through 
(specific parts of) the study materials before answering a new set of questions. 

The first set of questions was about the study approach of the course "Introduction to governmental law". 
The students got the opportunity to go through the course during about 5 minutes to refresh their knowledge. 
The interviewers suggested that the structure page and the introduction could be very helpful. 

The second set of questions dealed with the study approach of a learning unit the students themselves could 
chose. The students could go also through this learning unit 

The last set of questions was about learning unit 20. The study approach of this learning unit and the use of 
ESD when studying this learning unit was questioned. Beforehand the students got also the opportunity to go 
through this learning unit 

The interviewers went through the personal study materials of the students during the interview and used a 
separate checklist to gather information about the way students had 'personalised' their written course 
mat^als (marking, underlining, arrows, renumbering, written summaries, symbols, etc.). 

3.5 From interview variables to research variables 

Since the interview was based on a rigid protocol and structure, the researchers could relatively easily 
assemble the variety of answers for each question in order to structure the information. A codebook (176 
pages) was compiled that described how the ifidividual student answers were to be coded in an unambiguous 
way. The complete interview resulted in a data set of 499 interview variables. This data set was entered 
into the computer to be processed with the SPSS"^ statistical package. 
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The 499 interview variables were in a next step reduced to 191 research variables by combining, regrouping 
and receding. We give an example : students were asked three questions whether or not they had 
experience- with (1) studying at a distance 'AFSTERVA'(= AFSTandsonderwijs ERVAring: the dutch 
shortening of experience with distance education) (2) studying at the Dutch Open university 'OUERVA'(= 
OU ERVAring: Dutch for experience with studying at the Open university) and (3) the number of Ou- 
courses they already studied 'NCURSUS'(= N CURSUSsen: Dutch for number of courses). The answers to 
these three questions were recoded in order to compute the variable 'experience' : 

recode ctfsterva ouerva (1=1) (2=0) (9=sysims)l ncursus (9=0). 
coittpute experience = qfsterva + ouerva + ncursus. 

The number of categories for this new variable was 7. Analysis of the frequency table for this new variable 
helped to reduce the 7 categories to 3 by the following rule : 
recode experience (0=1) (1=1) (2=1) (3=2) (4=2) (6=3) (7=3). 
These new values are labelled : l=low, 2=medium, 3=high. 

To argument this receding and computing process, the researchers have introduced - for some variables - 
rationales about the nature of cognitive functioning. Three of these rationales refer to concqjts commonly 
found in literature about study regulation : surface level versus deep level studying (Marton & SSIjO, 1984), 
external versus internal control (Simons & Vermunt, 1986) and doep versus elaboiutive processing (Schmeck, 
1983). 

To judge whether the study behaviour of a student is to be judged as 'Surface level' or 'Deep level studying' 
a specific procedure was followed. During the interview, students had to respond to statements that describe 
the way one can study the course; e.g. : 

I read first the complete text book. 

I first read the text and die next time I mark parts of the text. 

I first read the work book completely. 

I start with the questions/tasks and next work step-by-step through the course. 
I study daily a prefixed number of pages. 

I read the learning objectives afterwards to control my learning process. 

I continue and repeat reading the content of a learning unit until I master the content 

I explicitly ad£^t to all 'study guidelines' in the text. 

The researchers discussed beforehand whether a behaviour was considered as an operational description of 
deep level versus surface level studying. The students indicated whether these statements did or did not 
reflect their own personal study behaviour. Agreeing with a 'surface level' operationalisation resulted in a 
'0' score. Agreeing with a 'deep level' operational description resulted in a '1' score. Summing the scores 
for the whole set of statements resulted in a new score for the new variable 'general learning strategy'^ The 
value for this new variables ranged from 6 to 25. Receding resulted in three categories (6 thru 8 = 1; 9 thru 
14 = 2; 15 thru 25 = 3). The 1-category refers to a rather surface level type of general learning strategy, 3 
refers to a deep level approach and 2 refers to a combination of both approaches. One can find, a detailed 
transformation procedure of this variable in appendix 2. 

Judging whether a complex behaviour or orientation was externally or internally controlled, depended - 
comparable to the procedure outlined above - on the sum of scores for a set or subset of statements. Again, 
students could agree or disagree with these statements : 

I make my own summaries. 

I add my own schemes. 

I underline text parts. 

I test the mastery of the learning content myself. 

I use especially my own annotations when studying. 

I rather use the study guidelines in the workbook to direct my study process. 
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Care was taken to 'correct' scoring if students did not respond to a specific statement. 



The answers to these questions were receded in order to compute the variable 'INTERN'. The documented 
transformation procedure of this variable can also be find in appendix 2. 

Deep versus elaborative processing refers to respectively making relations within the course material or 
between the course material and personal experiences. To construct this new variable the answers to the 
question which kind of relations the students make were receded. 

In receding and computing the research variables, the number of categories for each variable were limited. 
Since we projected to use multi-way frequency analysis, the numb^ of categories for each variable had to 
remain restricted, taking account of the small sample size. In a limited number of cases, too many missing 
values made the use of certain variables irrelevant They were excluded firom the further analysis. One can 
fmd additional examples of variable transformations in appendix 2. The complete reduction and receding 
procedure has been fully documented and more information can be obtained &om the audiors. 

3.6 Research variables 

In rehtion to each research variable we provide - between brackets - key words referring to the categories. 
We repeat that the categorisation is based on a receding and computing process that result in new variables. 
Frequency analysis of the scoreis for the new variable is used to look for relevant criteria to split up the 
scores in e.g. low, medium or high categories. 

The final list of research variables can be structured as follows : 

Descriptive variables about personal characteristics of the research sample 

Gender (man, woman) 

Age level (18-30, 30-40, \0-50, > 50) 

Educational level (low, medium, high) 

Variables related to the general orientation of students towards studying at the Open university 

Experience with distance education materials (Low, Medium, High) 

Locus of control (Extrinsic, Intrinsic) 

Level of intrinsic motivation (Low, Medium, High) 

Level of extrinsic motivation (Low, Medium, High) 

Motivation level (Low, Medium, High) 

Use of study guidance facilities (yes, no) 

Reliance on external support (Low, Medium, High) 

Variables in relation to studying law and the course "Introduction to governmental law" 
Prior knowledge in relation to the law domain (Low, Medium, High) 

Prior knowledge in relation to the course "Introduction to governmental law" (Low, Medium, High) 

Variables specifying the nature of cognitive functioning of students 

Orientation strategies at the start of the study (low, medium, high) 

Evolution in study speed during long term study process (change, no change) 

Development of metacognitive skills (yes, no) 

General learning strategy (surface level, deep level, combination) 

Self-controlled versus external controlled studying (external, internal, combination) 

Insight in main purpose of the course (via BSD, via external support, other ways) 

Average time spent studying (5-10, 12-19, 20-55 hours a week) 

Regularity of study-time (regular, irregular) 

Time planning (yes or no) 

Time spent per learning unit (same time, variation in time) 

Difficulties with switching between course materials (yes or no) 

Annotations in study books (yes or no) 

Problems witli looking up (enough trained, not enough trained) 

Source of overview (strongly, average ^; not by means of ESD/extemal sources) 
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Self-testing by means of ESD ( strongly, average or not) 
Self-testing by means of external source (strongly, average or not) 
Elaborative processing {deep versus elaborative ivccessing) 



Variables related to the performance of the students on the course exam 
Final score for the examination (< 6, 6-7, 8-10) 
Number of examination attempts (1, 2, > 2) 

Variables in relation to the use of ESD 

23 variables indicating the quantitative use of a specific ESD 

23 variables indicating deep level or surface level use of a specific ESD 

The evaluation of the de^ or surface level use of the ESD was based on tlie categorisation of the ftinctions 
and effects students attributed to each specific ESD : 

Statements that were categorized as deep level or surface level were of course also used to score for the 
variable "use of ESD". 
surface level use : "I did read it" 

"I noticed it" 

"I look at the indications for prior knowledge and take them for granted." 

"I have only scanned that text for a few seconds." 
deep level use : "I use this to look for othtir examples." 

"I always solve the c/uestions and control my answer by 

using the feedback iiformation." 

"This device helps me to understand the text." 

"This information is helpfiil to get an overview." 

"I have tried to answer that question." 

"After reading the introduction I reread a part that was 

ambiguous to me." 
Indications for 'no use' of a certain support device : 

"I don't use or read that kind of information." 

"I always skip the summaries." 

"Is there a content page?!" 

Additional research variables were created which indicated the use, deep level use and surface level use of 
ESD at a more general level. Four clusters of ESD were defined : 

- orientating ESD (e.g. content page, introduction, ...) 

- planning ESD (e.g. learning objectives, content page, ...) 

- processing ESD (e.g. advance oiganizCT, study guidelines, ...) 

- testing ESD (e.g. learning objectives, examples, ...) 

For each cluster the use, deep level use and surface level use was computed. 

4. QUANTITATIVE ASPECTS OF ESD USE 



4.1 Descriptive statistics 

To check hypothesis 1 "Students make no use of ESD" and hypothesis 2 "There is no difference in the 
number of students making surface level or deep level use of tiie ESD" we used a descriptive statistics 
technique: the data in relation to the variables about the use and level of use of ESD are analyzed. 

Table 1 gives an overview of the use and appreciation of the ESD embedded in learning unit 20 of the study 
text. The variables are given in an descending order of use. 

The first column indicates the overall quantitative use of ESD; this is the proportion of students that indicate 
they use this specific ESD in one way or another. 
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The second and third column examin in more detail the use of ESD. Here a distinction is made between 
'surface level' and 'deep-level' use^ In the cells the percentages of students are given that made at least one 
statement that was an indication for surface or deep-level use. Note that in this way it is possible that a 
student uses a support device in both ways. 



Table 1 : The use of ESD 





1 


2 


3 


4 


S 


ESD 


% used by 
students 


% surface use 
by students 


% deep level 
use 


used as 

meant 

(0-3) 


appreciatica 


study guideline 


100% 










exan^le 


100% 


28% 


96% 


2,92 




scheme (1) 


96% 


25% 


88% 


2,44 




question (2) 


96% 


6% 


96% 


2,83 


2,83 


introduction 


92% 


48% 


91% 


2,43 




maigin text 


92% 


4% 


96% 


2,79 




reference to other learning unit: 


88% 


6% 


92% 


2,75 




scheme (2) 


88% 


11% 


83% 


2,17 




title 


84% 


22% 


83% 


2,68 




task 


84% 


28% 


77% 


2,00 




exan^le of correct aasweis (feedback) 


84% 


12% 


79% 


2,58 


2,67 


sununaiy 


84% 


16% 


80% 


2,27 


2,33 


learning objective 


80% 


24% 


65% 


2,17 


2,39 


text paragraph 


80% 


0% 


96% 


2,68 




pre-question 


80% 


16% 


81% 


2,48 




question (1) 


80% . 


19% 


90% 


2,14 




font types (italics) 


80% 


10% 


76% 


2,52 




advance organizer 


80% 


65% 


70% 


2,13 




additional rematk 


16% 


41% 


67% 


2,33 




indications about prior knowledge 


68%, 


37% 


56% 


2,13 


2,65 


small printed text 


64% 


47% 


65% 


2,30 


1,82 


content page 


60% 


25% 


54% 


1,73 




facultative activity 


28% 


24% 


30% 


1,60 





The fourth column gives an indication whether the students used tfie ESD in concordance with our 
theoretical assumptions. A score '3' means that all students use an ESD exactly as expected, a '1' score 
means that no subjects use the support device as expected. Here a correction was made for the total number 
of possible ways to use an ESD. 



Because one can only use or not use a 'study guideline', no further questions were asked about this support 
device. This means that the cells 2, 3 and 4 for this particular row are empty. 
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Without such coirection the ESD that can be used in many ways are underestimated, since it is more likely 
that students do not use the ESD in al the possible ways. 

The last column gives information about the ^preciation of ESD. Due to time-constraints it was not possible 
to obtain information about the jq)preciation of all ESD during the interview. The scores vary from 1 to 3, 
reflecting a negative to a very positive appreciation of the ESD. 



4.2 Discussion 

The results in table 1 are, in our opinion, surprising. We did not envision such a high level of ESD use by 
the students. With the exception of 'facultative activity', all ESD are used by more than 60% of the 
students. 

If we use this information to check the first main null hypothesis "Students make no use of ESD", we can 
easily reject this hypothesis. 

When interpreting these results, we can wonder why ESD are so highly used. Several explanations can be 
put forward. It can be caused by the small sample size, the fact we selected law students, the specific course 
or the difficulty level of learning unit 20. 

As a consequence we have to interpret the results with caution. Although we tried to set up an ecologically 
valid and reliable research setup and tried to detect the 'real use' of ESD by students, it remains plausible 
that the student responses are not reliable. Their perceived and reported use and level of use of ESD can 
still differ firom what they actually do when studying. 

If we focus on the 'level' of ESD use, it is striking that students mention especially kinds of uses that are to 
be qualified as 'deep level'. This can bring us to the conclusion that 'if students use specific ESD, they use 
it in a 'deep level' way'. This suggests that we might also reject the second main null h>pothesis that states 
that there are no differences in deep level or surface use of ESD among students. 

We will derive no conclusions from the fourth collum ('used as meant') although we already perceive that 
ESD arc used as meant by the course developers. The only clear exceptions are 'content pages' and 
'facultative activities' (these variables score less than 2). We postpone a more detailed discussion of this 
topic to chapter 7. 

The last collum gives information about the appreciation of ESD. Unfortunately we only have information 
about 8 of the 23 ESD (due to time constrains). Nevertheless from these figures we can conclude that ESD 
are highly appreciated. The only exception concerns 'small print'. 

A final remark in relation to both the use and level of use of ESD is needed before we continue : The actual 
use of the ESD can be influenced by the quality of the specific implementation of the ESD in the study text 
If for instance 30% or 40% of the students do not use a specific ESD, is this because they consider this ESD 
as useless or is this an indication for a bad implementation of this distinctive ESD ? But it can also indicate 
that students never tend to use such an ESD, wether it is good or badly implemented. Analysis of tlie 
interviews showed for instance that 'the small printed text" was not used by most student but also not 
appreciated. Some students indicated that the used character font was too small to be readable. So it is quite 
possible that improving the font size would increase the use of this support device. 
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5. QUALITATIVE ASPECTS OF ESD USE 



S.l Analysis techniques 

In part 3.6 we gave an overview (5 clusters) of the research variables. Four of these clusteis describe 
individual student characteristics at a general (e.g. age. educational level, etc.) to a very specific ievel 
(mouvauon to study this course, specific study approach, etc.)*. Up to 30 variables can be used, when 
evaluating the hypotheses about the interrelation between general personal and cognitive student 
characteristics and the use of ESD. Multi-way frequency analysis is used to explore these interrelations. 

Interrelating 23 variables indicating the use of each specific ESD with 23 variables indicating the level of 
ESD use and 30 variables describing student characteristics, results in a matrix of 23 x 23 x 30 variables 
To explore this large set of variables data multi-way frequency analysis was performed to develop 
hierarchical loglinear models. But. due to the restricted number of students interviewed (N=25). the number 
of dimensions to be included in the model construction is to remain restricted in order to prevent a too large 
number of empty ceUs in the contingency tables (the "rule of five")'. To protect against empty cells, three- 
way frequency analysis was chosen as the most save way to explore the data and to develop models. Hie 
SPSS'" HILOG-procedure was used to evaluate the models. This procedure protects agai'-st violation of tfie 
rule of five by adding .5 to all observed cells. But still then, we have to remain cautious because observed 
and expected cell frequencies are smaU. The only solution left is to choose a severe, high criterion when 
judging the goodness-of-fit of models (e.g. a > 25f. 

FoUowing the SPSS'" HILOG-procedure. a backward stepping procedure was used, starting with the saturated 
model This model implies a perfect fit of the observed and the expected frequencies. The term (simple or 
multiple) that is least helpfiil to the model is deleted first, foUowed by assesst>jent of the remaining terms of 
the same order. In the SPSS'^-procedure. partial effects of individual effects are reported. Effects are deleted 
if they are unambiguously non significanL Backward stepwise deletion continues until the criterion for 
significance of fit is reached and the results adapt to the rule of parsimony. 



5.2 Results 

5.2.1 Student characteristics and ESD use 

Table 2 summarizes the results of the multi-way frequency analysis of the models where student 
characteristics are related to ESD use (each ESD seperately and next clusters of ESD). A remaric to be made 
IS that when we comprise the variables 'use of ESD' and 'deep level use of ESD' in one model, both 
variables interact to this extent that they suppress possible other interactions between the variables. This is 
not strange if we look at the results of table 1: an ESD which is used is almost always used at a deep level. 
Therefore, only 'deep level use of ESD' is incorporated in the models when the interrelations with student 
characteristics are explored. 

The first column describes the variables entered in the saturated model. The second column shows the 
number of ceUs when combining the numbCT of categories for each variable in the model. The third column 
indicates the significant, unambiguous effect(s) that remain in the model after stepwise deletion of ambiguous 
effects. The next columns shows the respective L? value, the degrees of freedom and significance level. The 
last column is a symboUc indication about the direction of the interrelation that is present in the significant 
effects. Take care, the table only contains information about models that result in 'significant' unambiguous 
interaction-effects. 



Quantitative infoimation about these student characteristics can be found in ^ndix 1 of this text where we give 
overview of frequency tables in relation to most research variables. 

The "mle of five" states that ccU frequencies in the table all are greater than 1 and that no more than 20% arc less than 5. 

Tabachnik & Fidell (1989. p.270) discuss in great detail the choice of a significance level when considering the sample si 
and the nature of the problem. 
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Table 2: Analysis of the hierarchical loglinear analysis 



variables in model 


dimen- 
sions 


significant unambiguoua 
eOects 




df 


p-valus 




gender 

deep level use of self-test ESD 

deep level use of ESD for planning 


2»3»3 


gender * 

deep level use of planning 
ESD 


12.1 


12 




* 


gender 

deep level use of orientating BSD 
deep level use of processing BSD 


2»3»3 


gendor • 

deep level use of 
orienUttog ESD 
gender * 

deep level use of 
processing ESD 


6.9 


8 


JSSO 


* 
* 


educational level 
motivation rating 


3»3»3 


educational level • 
deep level use 


7.4 


12 


jm 


* 


prior knowledge of governmental law 
age 

deep level use of testing ESD 


3«4«3 


prior knowledge of 
govemmenta) law * 
deep level use of totting 
ESD 


9.6 


18 


JI47 




use of etudy guidance 
gender 

deep level use of testing ESD 


2«2»3 


use of study guidance * 
deep level use of testing 
ESD 


3.3 


8 




* 



When looking for individual student characteristics that are related to the deqp level use of ESD, 



following significant interaction effects : 
The variable "gender" interacts significantly with : 

- deep level use of planning ESD 

- deep level use of orientating ESD 

- deep level use of processing ESD 

The variable "educational level" interacts significantly with: 

- deep level use of ESD 

The variable "prior knowledge of governmental law" interacts significantly with: 

- deep level use of testing ESD 

The variable "use of study guidance" interacts significantly vnth: 

- deep level use of testing ESD 



These results can be interpreted as: . t.ot^ j i i 

women are more likely to use planning, orientating and processing ESD at a deep level 
highly educated people iare more likely to use ESD at a deep level 
students with low prior knowledge of governmental law are more likely to use testmg ESD 
at a deep level , 
students' who use study guidance are more likely to use testmg ESD at a deep level 

Variables in the models that showed only main-effects are : 

- motivation 

- experience vnth distance education 

- elaborative versus within processing 

- evolution in study approach 

- regularity in study time 

- orientating 

- development of metacognitive skills 

Variables in models related to the use of ESD that showed no effects are : 

- reliance on external support 

- age 

- overview by means of ESD 

- overview by means of external source 

13 

o .. 22 



- switching between three books in the course 

- development of metacogniuve skills 

- development of metacognitive skDls (general/related to the law domain) 

- prior laiowledge of law 



5.2.2 Looking for 'types' of students using ESD 

In table 2 only interactions between use of ESD and general or cognitive student characteristics are 
displayed. We have also explored the interrelations wthin the cluster of general and cognitive student 
characteristics to get a more clear picture of the 'type(s)' of student which use ESD. In table 3 some 
significant results of this analysis are displayed. 

Table 3: Analysis of the hierarchical logUnear analysis between general and cognitive student characteristics 



variables in model 


dimen- 
sions 


significant unambiguous 
eSscts 


L' 


df 


p-valua 




locus of control (internal) 
gender 

educational level 


3«2«3 


gender • 

locus of control (internal) 


7.2 


10 


.70S 


+ 


gender 

use of study guidance 

prior IsnoTTledge of governmental law 


2»3«3 


gender • 

use of study guidance 


4.7 


6 


JS75 




intrinsic motivation 

prior knowledge of law 

prior knowledge of govemmental law 


3«3«3 


prior knowledge of law * 
intrinsic motivation 
prior knowledge of 
govemmental law * 
intrinsic motivation 


8.0 


12 


.768 


+ 
+ 


general learning strategy (deep) 

prior knowledge of law 

prior knowledge of govemmental law 


3«3«3 


prior knowledge of law * 
general learning strategy . 


105 


16 


.842 


+ 


annotations (oboervad by the student) 
general learning strategy (deep) 
use of study guidance 


2»3«3 


annotations (observed by 

the etudent) * 

use of study guidance 


2.6 


6 


.883 


+ 


atmotatlons (observed by the studenO 
annotations (observed by the interviewer) 
general learning strategy (deep) 


2«3»3 


annotations (observed by 
the student) • 
annotations (observed by 
interviewer) 


9.1 


10 


.621 


+ 


age 

prior knowledge of law 

prior knowledge of govemmental law 


4«3«3 


prior knowledge of law * 
prior knowledge of 
governmental law 


11.7 


18 


.804 


+ 



When looking at the general student characteristics we find the following interesting interactions in table 3: 
The variable "gender" interacts significantly with: 

- locus of control 

- use of study guidance 

The variable "prior knowledge of law" interacts significantly with: 

- intrinsic motivation 

- general learning strategy 

The variable "prior knowledge of govemmental law" interacts significantly with: 

- intrinsic motivation 

The variable "annotations (observed by the student)" interacts significantly with: 

- use of study guidance 

This last intraaction isn't surprising, because students are trained during study guidance meetings in making 
annotations. 

The last two interactions: 

- annotations (observed by the student) by annotations (observed by interviewer) 

- prior knowledge of law by prior knowledge of govemmental law 

are not interesting for our research questions, but they can be regarded as a control on our interview. 
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S.3 Discussion 



Before we discuss the results we should make the ranark that we have not explored all possible interactions 
because of the vast amount of variables in our investigation. So the interactions that are presented here 
should be seen as a minimal set of possible interrelations. 

On the base of the results of the hierarchical log-linear analysis we can test the hypotheses: 

1) There is no interaction between general student characteristics and the use of ESD. 

2) There is no interaction between cognitive student characteristics and the use of ESD. 

The first null hypothesis can be rejected: there are interrelations between general student characteristics and 
the use of ESD : 

- gender and deep level use of planning ESD 

- gender and deep level use of orientating ESD 

- gender and deep leve^ use of processing ESD 

- educational level and deep level use of ESD 

- prior knowledge of governmental law and deep level use of testing ESD 

- use of study guidance and deep level use of testing ESD 

The second null hypothesis cannot be rejected. This means that no cognitive student charactCTistics are 
retained in the models. 

Our findings are in line with and enrich earlier findings of Boon et al (1991b, p.l7) when they analysed the 
interrelation between student characteristics and course characteristics with a more general Ou-population 
(N=1661). Their analysis revealed (1) that 79% of the students use structuring elements of the course' 
regularily or very often and (2) that student characteristics such as 'lower educational level', 'having no 
study plan outside the Ou-context', etc.. are related to making more use of structuring course elements. 

We can question why only "gender", "educational level", "prior knowledge of governmental law" and "use of 
study guidahce" are retained as having an interaction effect with the use of ESD in the models evaluated? 
First, we can question our research setup. The restricted number of students inhibits reaching significant 
effects. If we review the data in the frequency tables of £^pendix 1, we find that in general the 25 students 
in our sample are not extremely different in relation to for instance motivation level, prior experiences with 
distance education, prior experiences with the law domain, reliance on external support, general learning 
^proach, orientating, etc. The characteristics of our sample influence as a result the extent to which 
differences pertain in study behaviour and the resulting use of ESD. 

Second, tiie fact tiiat all students use tiie ESD to a high extent makes it - a priori - difficult to find 
differences in use of the ESD. 

To get a more clear picture of tiie kind of stiidents which use ESD it is wortii looking at tiie results of tiie 
multi-way frequency analysis of the models about 'types' of students. We find for instance interactions 
between gender and prior knowledge of governmental law on the one hand and other personal charactOTstics 
on tiie otiier hand : gender interacts with locus of control (intern) and use of study guidance; prior knowledge 
of governmental law interacts with intiinsic motivation. We didn't find interactions between educational level 
and other personal or cognitive characteristics. Looking at tiie crosstabs (Appendix 3) shows anyhow tiiat 
there is a positive relation between educational level and prior knowledge of law, prior knowledge of 
governmental law, intiinsic motivation and general learning approach. From these four variables prior 
knowledge of law and prior knowledge of governmental law are most related to educational level. Pri6r 
knowledge of law itself interacts witii intrinsic motivation and a deep level learning strategy. 
So when considering tiie cluster of general and cognitive student characteristics which interact we get tfie 
following picture: especially highly educated (and intrinsic motivated) stiidents witii many study experience 
and little prior knowledge make deep level use of ESD. 



They distinguished 8 structuring elements : structure pages, learning objectives, questions and feedback to tyhe questions, 
example examination, specifal requirements, supoort facilities outside the course and media add-ons. 



IS 



24 



6. STUDY SUCCESS IN RELATION TO USE OF ESD 



6.1 Building moiels 

We also gathered infonnation about stud;' success : final examination scores (score) and number of 
examination attempts (attempts). We can use this information to build additional models that are helpful to 
describe/explain ESD use. Table 4 contains the results of the multi-firequenoy analysis of the models where 
study success is related to the use of ESD and other student characteristics. 

The first column describes the variables in the saturated model. The second column shows the number of 
cells when combining the number of categories for each variable in the model. The third column shows the 
significant, unambiguous effects that remain in the model after stepwise deletion of ambiguous 'Effects. The 
next columns shows the respective I? value, the degrees of fieedom and significance level The last column 
is a symbolic indication about the direction of the interrelation that is present in the significant effects. 

Table 4: peifonnance of students in relation to the use of ESD and student cbatacteristics 



variables in model 


dimensions 


sIgniScant 
unambiguous eCEects 




df 


p-value 




development of metacognitive 

akilli 

score 

deep level use of ESD 


2»3»3 


score * 

deep level use of BSD 


4 


8 


.858 


+ 


attempts 

reliance on external support 
deep level uce of BSD 


3»3»3 


attempts * deep level 
use of BSD 




18 


.678 


- 


attempts 
•core 

development of metacognitive 
•idlls 


3«3«2 


attempts * score 


1£ 


8 






educational level 

prior knowledge of governmental 

law 

attempts 


3»3»3 


attempts * educational 
level 


9.4 


16 


Jiii 




educational level 
score 

prior knowledge of governmental 
law 


3«3«3 


educational level * 
score 


12.3 


16 


.720 


+ 


experience with distance 

education 

score 

prior knowledge of law 


3»3«3 


experience with 
distance education * 
score 


19.6 


18 


.860 


+ 


orientating 
score 

prior knowledge of law 


3»3»3 


orientating • 
score 


18.8 


18 


.405 


+ 


switching between course 

materials 

score 

prior knowledge of law 


2«3«3 


switching between 
course materials * 
score 

1 


16.4 


18 


.633 


+ 



Study succes versus use of ESD : 

The variable "score on the examination" interacts significantly with 

- deep level use of ESD 

The variable "attempts" interacts significantly with 

- de level use of ESD 

Student characteristics versus study success : 
The variable "attempts" interacts significantly with 

- score on the examination 

- educational level 

Thf variable "score on the examination" interacts significantly with 

- educational level 
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- experience with distance education 

- switching belvveen the course materials 

- orientating 

These results can be interpreted as follows : 

- students with a high examination score are more likely to use ESD at a deep level 

- students with a low number of examination attempts are more likely to use ESD at a deep level 

- students with a low number of examination attempts are more likely to acquire a high examination score 

- students with a low number of examination attempts are more likely to have a high educational level 

- students with a high examination score are more likely to have a high educational level 

- students with a high examination score are more likely to have more experience with distance education 

- students with a high examination score are more likely to switch properly between the course materials 

- students with a high examination score are more likely to orientate. 

62 Discussion 

Two main conclusions can be drawn £rom these results. 

The first conclusion is that students with a high examination score and with less examination attempts use 
ESD at a deep level. 

The second conclusion refers to the type of students acquiring a high examination score and do this with less 
examinadon attempts. The following picture arises: they are highly educated, diey have experience with 
distance education, they switch property between the course materials and they orientate more easily. 
Strilring is the fact that "educational level" is related to all fte variables that are important in relation to the 
use of ESD. Again students with a higher educational level seem to use ESD at a deep level and score 
consequently higher on the examination after less examination attmpts. 

Again, we have to repeat that caution is needed when interpreting or generalising these results. Our specific 
student sample and our research procedure can have influenced these particular ourcomes. 



7. SPONTANEOUS VERSUS THEORETICAL USE OF ESD 

7.1 Comparing spontaneous use and theoretically expected use of ESD 

In chapter 4.1 we already anticipated on this topic. In this section we present additional information. 
First we re[eat that the theoretical functions and effects of ESD gave been derived from our theory. Second, 
the functions and effects of ESd as perceived by the students have been derived from the interviews. In 
order to obtain this information we induced from the students spontaneous reactions to a general open ended 
question. Next, we posed specific questions based upon our theory (e.g. 'has the content page been helpful 
to you to orientate in the study material?') to check whether they agreed with the functions and effects 
recited. 

Table 5 sunmiarizes the result^. "ITie first collumn gives a list of all ESd we distinguish in our theory. 

Next, the other collums refelct all the potential fimctions/effects fliat are expected. 

The percentage of students that 'spontaneously' reported a certain effect for a certain support device 

is indicated in the cells. For example, if we look at row 1 and column 15 we can see that 15.8% of the valid 

cases indicated spontaneously that indications about required prior knowledge have a facilitating effect on 

understanding the task environment. 

The content of some cells contains an asterix (*). This means that students indicated a function although it 
was not theoretically expected. '0%' means that no student spontaneously mentioned a function, although this 
was expected. 



' The first colliun lists all the ESD we discern in our theory. Since the actual study only focuses on a subset of the general set 

of ESD. the collums for certain ESD are not fully documented 
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7.2 Discussion 



The results in table 5 reveal that if there is m agreement between theoretically expected and spontaneous 
mentioned functions, this percentage is low: only for 3 E?D more than 40% of the students mention the 
functions that are theoretically expected. 

Another conclusion that can be derived from table 5 is that many theoretical functions of ESD are not 
mentioned spontaneously by students (cells with '0 %')• When we confront these results with the results of 
table 1 we see anyhow that if students are explicitly asked to indicate functions/effects, they indicate that 
they use the ESD as meant (see column 4 of table 1). These results could imply that students are not aware 
of specific functions/effects of ESD. It is also possible that students are not able to explain why a certain 
embedded support device improves text comprehensibility or facilitates the study process. 
Also interesting is that the students indicate functions spontaneously that are not theoretically expected (this 
is represented by the scores with an asterix (*). Students seem to ascribe personal functions/effects to ESD. 

Does this imply that we cannot validate our theory about functions and effects of ESD ? 

In our opinion, the research results do not contradict our theory. They rather reval that, whatever functions 

or effects are assigned to certain ESD in written study materials, student; redefine the functions/effects in 

terms of their own study needs, study plan and cognitive functioning. Therefore, the assignmwit of fimctions 

or effects cannot be made in general terms, but has to take account of at least 'types' of students. 

The question can also be put forward whether it is relevant to 'leam' students to use specific ESD. We can 

refer e.g. to the research of Winne (1983) and de Jong & Simons (1988) who demonstrated how training in 

the use of ESD can result in a better 'supported' learning process. In our opinion the findings stress the 

importance of such 'training'. 

Finally we again remark that these results should be interpreted with caution. The ascription of the general 
and undetailed student answers to specific theoretically expected functions/effects was not an easy and 
unambiguous task. 
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8. GENERAL DISCUSSION 



First we will summarize and comment the overall research findings in relation to the use of ESD. Secondly 
we will link the main findings of our study to related research. Next, problems regarding our method of 
investigation are considered anci finally we focus on possible directions for future research. 

8.1 Putting things together : is the elaboration of written study materials worth the effort ? 
Implications for future elaboration of Ou-courses 

When we summarise the research results about the actual use of ESD we get the following picture : 
ESD are highly used: with the exception of the 'fecultative activity' all ESD are used by 
more than 60% of the students. 

The level of ESD use is to be qualified as deep level use. If students use ESD, they seem 
to use ESD at deep level. 

To get a picture of the 'type' of students which use ESD at deep level, we analysed interactions with general 
and cognitive student characteristics. To get a more clear picture we also exploded the interrelations between 
the set of personal and cognitive characteristics. We found significant interactions between gen^ stud^t 
characteristics and the deep level use of ESD: gender, educational level, prior knowledge of governmental 
law and use of study guidance. The interactions between cognitive student characteristics and de^ level use 
of ESD were not significant When we analysed the cluster of student characteristics we obtained a more 
clear picture of the 'type' of students which seem to make de^ level use of ESD : especially highly 
educated and intrinsically motivated female students with distance education experience and little prior 
knowledge make deep level use of ESD. 

As to the possible relations between deep level use of ESD and study success it seems that students with a 
high score on the examination and needing less examination attempts make deep level use of ESD. 

The differences between spontaneously mentioned and theoretically expected functions and effects of ESD 
were striking. Many theoretically expected functions and effects are not or rather rarely mentioned by the 
students. Students indicate on the other hand fimctions and effects that are theoretically not expected. 
Students seem to ascribe their own functions and effects to ESD. 

Returning to our main research question we think we can conclude that the high investment in the design and 
elaboration of study materials is worth the effort. Students use ESD, even at a deq) level. The finding that 
students which use ESD at a deep level have a high examination score and need less examination attempts is 
an extra indication that it is worth the effort. 

The results concerning the interaction between general student characteristics and use of ESD indicate that 
student characteristics have to be taken into account when developing or presenting study materials. Study 
materials have to be flexible and adaptable to individual student needs or the needs of 'types' of students. 
The fact that students spontaneously ascribe their own functions and effects to ESD can be considered as an 
extra stress for fiirther flexibilisation. Moreover, it can be argued to train studwits to make use of ESD. 

8.2 The research results compared to earlier findings 

In chapter 5.3 we already linked our findings to other research set up at the Dutch open university. Our 
results are in line and confirm the results of earlier Ou-studies. But we can also relate the results of this 
study to the research results of three researches especially linked to the course "Introduction to governmental 

law" : 

1. The actual use of ESD by students in the course 'Introduction to governmental law': the opinion of 
tutors (Martens & Daal, 1992). 

2. The use of ESD in the course 'Introduction to governmental law'. (Martens et al, 1991). 

3. Evaluation of the course 'Introduction to governmental law' (van Meurs, 1987)'. 



' The three titles are translated from Dutch to English by the authors. 
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The first study to be contrasted with our own results is based upon the opinion of tutors. The conclusions of 
this research indicated that the abstraction level of the course is - in general - too high. The amount of 
examples is too restricted. 

Can tfiis be related to some the findings of the actual study ? When we look at the BSD 'examples' in table 
1, we perceive that all students make use of examples and this at a deep level. This can be expounded as an 
indication for the high need for examples. However, these figures cannot tell us whether there is a need for 
extra examples. 

The results bom a second earlier study are based on a review fiom an educational point of view. For this 
review we used a checklist developed by Valcke et al (1991). The conclusion of this checking was that some 
BSD are not adequately implemented. A typical illustration is ^ain the ESD 'examples'. Only 4% of the 
learning units reflect a fitting and proper use of 'examples'. When relating the findings from report 2 to the 
actual study we get the pattern displayed in table 6. In the first coUum are printed the BSD used at deep 
level by less than 80% of the students. In the second coUum the ESD that are not appropriately implemented 
(based on report 2) are displayed. The figure < 80% in the first coUum is misleading since - with the 
exception of 'facultative activities' - most ESD are still used by more than 60% of the students. 



Table 6: comparison of deep level use of ESD with scsne conclusions from report 2 



deep level use < 80% 


not correctly implemrated from an 
educational point of view 


content pages 


. "> 


indication about prior knowledge 


X 


study goals 




tasks 


X 


examples of correct answers 




additional remaiics 


X 


facultative activities' 


X 


text in italics 




small print 




advance otganizer 


X 



Due to the high amount of ESD use, we cannot easily derive conclusions from this table. If the amount of 
ESD use was extremely low for certain ESD, this could have meant that restricted ESD could be explained 
by a bad implementation. But since this prior condition is not true (cf. table 1) we cannot confirm the latter 
statement and reject more easily the suggestion that die actual use of BSD might be influenced by the 
specific (poor or restricted) implementation of the ESD. 

A third, earlier, study evaluated the general acceptance of the course by a sample of law students. 
The main conclusions from this evaluative research were : 

1. Students have problems with handling the textbook-workbook-reader construction. 

2. Students tend to foUow the suggested working method. 

3. In general the students are satisfied with the course. 

The first conclusion is in line with our findings, 13 students of the 25 students have problems with handling 
the three books. 

Also the second conclusion can be confirmed when looking at table 1 : alle 2S students use study guidelines. 
To compare conclusion 3 we have to look again at table 1. In most cases students are satisfied with the 
implementation of support devices. Of course this can only be said about the ESD from which we obtained 
'appreciative' information. 



'-' means that there ia no information available about this ESD. 
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8.3 Putting the results in the right perspective 

At some points during the discussion of the research results we aheady asked for caution w^en interpreting 
the results. We r^)eat here these considerations in order to put the resdts in the right perspective. 
We structure them by looking at th,e one hand to shortcomings in the research method adopted and on the 
other hand to shortcomings resulting fipom our research sample. 

When considering the alternative research methods for this research project (Valcke & Martens, 1992) 9 
methods for investigating ESD were discussed. It was concluded that the interview-method was to be 
choosen for this study but that there are also disadvantages linked to this choice : 

Students are not always conscient of their reading- and study process. Being aware of the 
use of ESD would demand from the students a functioning at different levels (a stiKly and a 
meta-study level), and the question is whether the cognitive load is not too high to expect 
this level of awareness. 

Analyzing the student's answers demands an interpretative activity of the researchers. Also 
the translation firom interpretations to variable categories implies a translation phase. These 
translation phases might affect the reliability and validity of the research data. 
Analyzing interviews is time consuming. The price to be paid for the high quality of the 
research data is a huge investment in time and energy to transcribe, to analyze and to 
categorize student answers. 

We cannot neglect the first disadvantage mentioned. We attempted to scaffold the retrograde introspective 
process by carefully building up the interview session and by making use of the students' study materials. 
The stepwise building up of questions (e.g. first asking spontaneous reactions before asking for specific 
responses to specific functions/effects) still cannot prevent the students bom giving unreliable or socially 
accq)table answers. 

Regarding the second disadvantage, we tried to overcome this problMn as much as possible by interviewing 
on the basis of a detailed interview protocol. The intCTView followed a structure that evolved from the 
general to the very specific (at ESD level) aspects of the individual learning process. This facilitated to check 
the comprehension of the questions by the students and the consistency of the student answers. We also 
repeat that the interview was structured by a rigid structure that linked question unambiguously to interview 
variables. It is our opinion that this resulted in 'translation' phases that did not afiect the reliability of the 
student answers. 

The third disadvantage 3 is again difficult to overcome. We reduced flie time load for administering, 
transcribing and analysising the interview by carefully prestructuiing the interview. But still under these 
conditions the set up was so time-consuming that the number of students in Ae sample had to remain 
restricted. 

Our research sample is - as said earlier - restricted. Although the 25 students interviewed form a 
representative sample from the Ou-law student population still other problems remain unsolved. We 
restricted for instance our study to students : 

- taking an introductory course; 

- studying taw; 

- taking examination. 



8.4 Future research 

Taking account of the former section, we have to stay decent when extrs^lating conclusions bom this 
research. Future research has to be set up to confirm/disconfirm our findings with other courses, other 
student populations and in different settings. Moreover, the research methodology has to be ad^ted to 
incorporate methods that result in reliable and objective student reactions. We briefly elaborate this issue : 

A methodological problem regarding our research setup is caused by the 'subconscious nature' of 
pans of the study process. It is difficult to get access to these processes by means of an interview. 
To enrich the reserach set-up a combination of research instruments could be developed, including 
for instance eye-movement registration, reading comprehension tests, etc. 
Our research was Unked to only one content domain: law. Even within this domain only one 
discipline was selected: governmental law. One can anyhow question whether the research results 
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are to be generalised to other domains and disciplines. Further research of use of ESD within other 
domains and disciplines is therefore needed. 

In this research some clear intwactions between student characteristics and use of ESD were found. 

More research is needed to explore these relationships : is it e.g. possible to separate in a better way 

'types' of students ? . 

Also the ways in which students can be trained in using ESD has to be explored in a systematic 

way. 

Further research can focus on preliminary implementations of the reserach findings. 
A first idea can be linked to flexible course development student characteristics should be taken 
more in account. An automated interactive learning envimment could be linked to a course. In this 
course, the student can opt for the incorporation of specific ESD. For instance the abstraction level 
of the course can be lowered by implementing more examples and by providing more text structure. 
Also the position and combination of ESD can be varied. Only after tiiis preliminary study phase in 
front of the computer a text version of the course is produced. 

Anotiier exasnple can result in a special course to train "study skills" for starting students. The 
individual differences found in this research support the rationale of such initiative. 



9. FINAL CONCLUSIONS 

In tfiis investigation we tried to answer tfie question : "Is tfie high investment in the design and elaboration 
of written study materials for use in a distance education setting, worth tfie effort ?". From tfie results of tfiis 
study we can conclude tfiat it is worth tfie effort. 

Several null hypotheses could be rejected: 
Students do use ESD. 

Moreover they use tfiem at deep level. Also important is the feet tfiat deep level use of ESD is 
related to higher examination scores and fewer examination attempts. 
There are interaction between the (deep level) use of ESD and general student characteristics. 
A specific null hypotfiesis could not be rejected : 

We did not find significant interactions between use of ESD and cognitive student characteristics. 

General student characteristics, such as gender, education level, prior knowledge of governmental law and 
use of study guidance play a role in tfie use of ESD. To get a better picture of tfie 'type' of students which 
use ESD at a deep level, we looked at interactions witfiin the large set of student characteristics. 
The 'type' of students making deep level use of ESD could be described as highly educated witfi study 
experience, a high examination score and few examination attempts. Students witfi litfle prior knowledge use 
testing ESD more at a deep level. 

Implications from tfiese results suggest that it might be necessary to take into account student characteristics 
when developing course materials and implementing ESD. 

Finally it was concluded tfiat tfiere is a difference between spontaneously mentioned fimctions/eflfects of ESD 
and tiieoretical expectations. Students mention tfieir own functions and effects. This can be caused by tfie 
fact tfiat students are not always aware of tfie fiinctions and effects of tfie ESD as tiiey are conceived by tfie 
course developer. 

This could imply that it is necessary to give students training in making use of ESD. 

An alternative explanation of the disagreement between tiieoretical and actual fimctions/effects of ESD can 

be found in tfie cJ5)ability of tfie students to escribe tfieir own functions and effects to ESD. 

This plausible alternative explanation, combined with tfie finding tfiat tfiere is an interaction between student 

characteristics and ESD use, puts forward extra arguments to develop and deliver Ou-courses in a flexible 

way. These conclusions are in line witfi current research activities at tfie Opci university where automated 

interactive learning environments are linked to Ou-courses. 

Finally, we repeat tfiat tiie research results have to be interpreted witii caution since particularities of tfie 
research setup (interview procedure and research sample) could have affected tfie quality of tfie research data 
inhibiting a generalisation of the research results to tfie general education Ou-practice. 
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APPENDIX 1 



OVERVIEW OF FREQUENCY TABLES IN RELATION TO THE RESEARCH 
VARIABLES 



A. Variables related to tbe general orientation of students tovrards studying at the Open university: 



Experience with distance education materials: 

Low 

Medium 

High 

Missing 



3 
9 
11 
2 



Locus of control: 
Intrinsic motivation 
Extrinsic motivation 
Missing 



24 
0 
1 



Level of intrinsic Motivation: 

Low 

Medium 

High 

Missing 



7 
13 
4 
1 



Level of extrinsic motivation: 

Low 

Medium 

High 

Missing 



6 
12 
6 
1 



Motivation level: 
Low 
Medium 
High 



1 
4 

20 



Use of study guidance facilities: 

Yes 

No 



16 

9 



Reliance on external support: 

Low 

Medium 

High 

Missing 



9 
7 
2 
7 



B. Variables in relation to studying lave and the course "introduction to governmental lavr" 



Prior knowledge in relation to the law domain: 

Low 

Medium 

High 



4 
13 
8 



Prior knowledge in relation to the course "Introduction to governmental law": 
Low 17 
Medium 4 
High 3 
Missing 1 
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C. Variables specifying the nature of cognitive functioning of students 



Orientation strategies at the start of the study: 

Low 10 

Medium 12 

High 1 

Missing 2 

Evolution in study speed during long term study process: 

Change g 

No change 17 



From the 8 students who indicate that there have been changes, 3 woriced at a slower rate and 5 worked 
faster at the end of the course. 



Development of metacognitive skills: 

Yes 1 1 

No 13 

3 students reported that they have developed skills that are typically related to the law domain (e.g. 'looking 
up lawtexts') and 8 students have developed more general study skills such as "planning", "thinking more 
logical", "making annotations". 



General learning strategy: 

Surface level approach 1 

Combination of both approaches 7 

Deep level approach U 

Missing g 

Self-controlled versus external controlled studying: 

External control 11 

External and internal control 10 

Internal control 4 



Insight in the main purpose of the course: 

Insight via ESD 

Insight via external support 

Other ways 

Missing 



9 
2 
4 
10 



Average time spent studying: 
5-10 hours 
12-19 hours 
20-55 hours 



7 
9 
9 



Regularity of study-time: 

Regular 

Irregular 



Planning: 
Study-plan 
No study-plan 
Missing 



18 
7 



17 
7 
1 



Time spent per learning unit: 
The same time per unit 
Variation in time per unit 



7 
18 
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Difficulties with switching between course materials: 

Yes 

No 



12 
13 



Annotations in study-books: 

Yes 16 

No 9 

Problems with looking up: 

Enough trained 18 

Not enough trained 6 

Missing 1 

Source of overview: 

Overview strongly via ESD 3 

Average overview via ESD 8 

Overview not via ESD 10 

Missing 4 

Overview strongly via external source 1 

Average overview via external source 15 

Overview not via external source 5 

Missing 4 

Self-testing by means of ESD: 

Testing strongly by means of ESD 6 

Average testing by means of ESD 17 

No testing by means of ESD 1 

Missing 1 



Self-testing by means of external source: 

Testing strongly via external source 3 

Average 10 

Testing not via external source H 

Missing 1 

Elaborative versus deep {or within-course) processing: 

Deep processing 8 

Elaborative processing 6 

Missing H 



D. Variables in relation to the performance of the students on the course exam 

Score for the examination on the course "Introduction to governmental law" 
<6 5 
6-7 11 
8-10 9 

Number of examination attempts 

1 11 

2 ^ 
>2 ^ 
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APPENDIX 2 EXAMPLES OF TRANSFORMATIONS OF VARIABLES 



- Students were asked whether or not they had experience with law 'ERVRW, If they had, they were asked 
to describe fliis experience 'OMERVRW. The answers to this questions were receded to compute the 
variable 'VOORKREC, the dutch shortening of prior knowledge law: 

recode ervrw omervrwl to omervrw4 (1=1) (2=0) (9=0). 

compute voorkrec=ervrw + omervrwl + omervrwl + omervrw3 + omervrw4. 
The numbCT of categories for this new variable was 5. Analysis of the firequency table for this new variable 
helped to reduce the 5 categories to three by the following rule: 
voorkrec (0=1) (2=2) (3=3) (5=3) 

These new values are labelled: 1 = low, 2 = medium, 3 = high. 

- Students were asked what their study approach was by the start of their study 'STRBEGI'. The study 
approaches were reduced to orientating or no orientating. The new variable 'ORIENTAT' was created as 
foUows: 

recode strbegil to strbegB strbegiS to strbegi? (1=1) (2=0)1 strbegi4 (2=1) (1=0). 

compute orientat= strbegil + strbegi2 + strbegi3 + strbegi4 + strbegiS + strbegi6 + strbegi?. 
The number of categories for this new variable was 6. These 6 categories were reduced to three by the 
following rule: 

orientat (0=1) (2=1) (3=2) (6=3) 

Thes? new values are labeUe± 1 = no or lide orientating, 2 = medium orientating, 3 = much orientating. 



- All variables which referred to study approach, such as 'STRALG' (study ^proach in general) 
'STRATLEH' (study approach of self chosOT leamiiig unit) 'STRATrWI'(study approach of learning unit 
20) etc, were receded: 1 = deep level approach, 0 = surface level approach. Only the variable 
'WEERGAVE' was receded as follows: 0 = deep level approach, -1 = surface level approach. This because 
this variable referred, in our opinion, less to the study jq)proach compared with the other variables. The new 
variable 'DIEPTE', the dutch shortening of deep, was created as foUows: 

recode stralgl stralg9 tijdleh (1=0) (2=1)1 stralg2 stralgS stralg6 stralg? stralgS stralglO stralgll 
stralgl2 stralgl3 plamlg (1=1) (2=0)1 overzich bedoelle typedoel relatie overztwi (1=2) (2=0)1 
stratleh (01=0) (02=1) (03=sysmis) (04=sysmis) (05=1) (06=1) (08=1) (09=0) (10=1) (11=1) 
(13=1) (14=sysmis)l weergave (01=0) (02=0) (03=0) (04=0) (06=-l) (07=0) (08=0)1 strattwi 
(01=0) (02=1) (04=sysms) (05=1) (06=1) (08=1) (09=0) (10=1) (11=1) (13=1) (16=1) (18=1). 
compute diepte=planalg + tijdleh + stralgl + stralg9 + stralg2 + stralg5 + stralg6 + stralg? + 
stralg8 + StralglO + stralgll + stralgl 2 + stralg 13 + overzich + bedoelle + typedoel + relatie + 
overztwi+ stratleh + weergave + strattwi. 
The number of categories for this new variable was 25. These 25 categories were reduced to three by the 
following rule: 

diepte (6 thru 8=1) (9 thru 14=2) (15 thru 25=3) 

These new values are labelled: 1 = surface level approach, 2 = mixed J^proach, 3 = deep level approach. 



- A set of variables which refer to the cognitive functioning of students were reduced to externally or 
internally controlled study behavior. The variables were receded as foHows: intern controlled study behavior 
= 1; externally controlled study behavior = 0. The new variable 'INTERN' was created as foUows: 
recode stralgM (1=0) (2=1)1 samenv stralgl5 schema strepen overign toetsen (1=1) 
(2=0)1 eighulp (1=1) (2=0) (5=05). 

compute intern= stralgM + samenv + stralgl5 + schema + strepen + overign + 
toetsen + eighulp. 

The number of categories for this new variable was 8. These 8 categories were reduced to three by the 
following rule: 

intern (2 thru 4=1) (5 thr 6=2) (7 thru 8=3). 

These new values are labelled: 1 = externally controlled, 2 = mixed controlled, 3 = internally controlled. 




APPENDIX 3 CROSSTABS OF THE VARIABLE "EDUCATIONAL LEVEL" WITH OTHER 
GENERAL AND COGNTITVE STUDENT CHARACTERISTICS 



Educational level by prior knowledge of law 

Prior knowledge of law 





Low 


Medium 


High 


Educational level 








Low 






1 


Medium 


3 


8 


1 


High 


1 


5 


6 



Educational level by prior knowledge of governmental law 

Prior knowledge of governmental law 
Low Medium High 



Educational level 








Low 


1 






Medium 


10 


1 




High 


6 


3 


3 



Number of Missing Observations: 1 
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Educational level by intrinsic motivation 

Intrinsic motivation 
Low Medium 



Educational level 

Low 

Medium 

ffigh 



4 
3 



Number of Missing Observations: 1 



1 

6 
6 



High 



Educational level by general learning approach 

General learning approach 
Low Medium 



Educational level 

Low 

Medium 

High 



4 
3 



High 



4 
7 



Number of Missing Observations: 6 
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